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SUMMARY 


Successful  wartime  demand  rate  forecasting  depends  heavily  on 
historical  data  from  which  to  project  those  demands.  Historically, 
peacetime  data  are  used  to  make  these  projections.  Previous  analysis 
suggests  that  continued  use  of  this  approach  may  provide  misleading 
conclusions. 

For  several  years  the  Air  Force  Human  Resources  Laboratory  has 
investigated  the  problem  of  using  actual  combat  data  in  this 
analysis.  This  paper  addresses  some  of  the  lessons  learned  from  this 
investigation.  Both  technical  and  contract  management  issues  are 
discussed.  Technical  areas  include  how  the  data  should  be  collected, 
normalizing  the  data  for  analysis,  and  specific  concerns  with  using 
combat  data.  Even  though  the  procurement  process  is  well  defined, 
several  actions  are  recommended  to  assist  the  contract  manager  in 
the  administration  of  a  contract. 
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PREFACE 


This  paper  summarizes  some  of  the  lessons  learned  over  the  past 
5  years  while  developing  a  combat  data  base  from  actual  combat 
operations.  These  lessons  were  learned  from  various  contracts  and 
efforts  conducted  by  the  Air  Force  Human  Resources  Laboratory 
(AFHRL).  These  research  and  development  efforts  were  part  of  an 
overall  Logistics  and  Human  Factors  Division  (AFHRL/LR)  approach  to 
forecast  wartime  demand  rates  using  combat  data  instead  of  peacetime 
data.  The  final  product  is  also  usable  by  analysts  in  the 
operations,  maintenance, and  combat  modeling  arenas. 
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COMBAT  DATA  BASE  DEVELOPMENT 
LESSONS  LEARNED 

I.  INTRODUCTION 

There  is  a  need  for  the  United  States  Air  Force  (USAF)  to 
develop  an  analytical  basis  for  forecasting  requirements  necessary  to 
maintain  adequate  sorties  during  wartime  conditions.  Successful  and 
efficient  sortie  generation  depends  heavily  on  data:  operations, 
intelligence,  threat,  logistics  and  maintenance  data.  Peacetime  data 
are  readily  available,  but  complete  and  accurate  data  for  wartime 
operations  are  rarely  available.  The  underlying  problem  of  meeting 
this  need  is  obtaining  accurate  and  valid  data  in  a  usable  form  to 
conduct  analysis.  This  research  effort  was  directed  at  resolving 
these  problems. 

The  objectives  of  this  research  and  development  (R&D)  effort 
were:  (a)  the  collection  of  integrated  aircraft  maintenance, 
logistics,  threat,  and  operations  data  from  an  actual  conventional 
war  with  opposing  Soviet  built  aircraft;  (b)  the  collection  of 
identical  data  from  a  peacetime  environment  for  comparison;  (c)  the 
development  of  an  automated  data  retrieval  and  analysis  system  to 
manipulate  the  data;  (d)  the  development  of  normalization  techniques 
to  relate  the  data  to  present  US  aircraft;  and  (e)  the  analysis  of 
the  data  base  to  identify  anomilies. 

Since  the  USAF  has  not  been  involved  in  a  sustained  conflict 
since  the  Vietnam  War,  the  required  data  was  only  available  from  a 
foreign  source.  This  greatly  complicated  the  contracting  process. 
The  contract  strategy  used  was  a  classified  sole  source  contract  with 
a  US  company.  However,  the  majority  of  the  work  was  subcontracted  to 
a  foreign  company  with  the  required  expertise  and  the  necessary 
clearances . 
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This  paper  will  address  some  of  the  lessons  learned  from  this 
effort  including  several  related  contracting  issues.  Development  of 
a  combat  data  base  which  integrates  various  data  areas  is  extremely 
difficult.  Even  though  the  contracting  process  is  well  defined,  the 
contract  monitor's  responsibilities  for  contract  management  includes 
many  time-consuming  administration  tasks. 

II.  TECHNICAL  ISSUES 

Data  Collection  Techniques 

Combat  data  collection  should  not  be  limited  to  quickly  getting 
the  desired  data  for  a  specific  use.  Combat  data  collection  is  an 
expensive  process  and  must  be  approached  systematically  so  that  the 
data  can  have  multiple  uses.  The  following  data  collection 
techniques  should  be  followed  during  actual  combat  situations,  major 
exercises,  operational  readiness  inspections  (ORI),  or  drills. 

1.  Identify  required  data:  Extensive  effort  must  be  taken  to 
identify  not  only  the  immediate  data  requirements  from  planned  data 
collection  activities  but  also  future  requirements.  Once  the  data 
collection  activity  is  completed,  it  is  too  late  to  add  data 
requirements.  Also,  if  possible,  consider  the  data  requirements  of 
other  likely  user  organizations.  This  increases  the  useability  of 
the  data  base. 

2.  Identify  available  data:  All  available  sources  of  the 
required  data  must  be  reviewed.  During  any  data  collection  activity, 
the  validity  and  accuracy  of  each  data  element  identified  must  be 
examined.  This  examinination  should  include  when  the  data  were 
collected,  where  they  were  collected,  how  they  were  recorded,  and  who 
recorded  them.  For  example,  during  exercises,  data  can  be  collected 


by  an  assessor  or  technican  participating  in  the  exercise  or  by 
independent  data  collectors.  Personnel  participating  in  the  exercise 
are  normally  in  an  ideal  position  to  provide  the  most  accurate  data. 
However,  the  pressures  of  the  situation,  such  as  time,  mission 
oriented  protective  posture  (MOPP)  gear,  and  evaluations,  can  distort 
the  reliability  of  such  data.  Hence,  the  useability  of  data 
collected  from  existing  data  reporting  forms  must  be  reviewed. 
Independent  data  collectors  record  data  from  their  obsevations.  They 
should  not  interfere  with  the  exercise.  If  they  do  not  see  it,  they 
can  not  record  it.  The  pros  and  cons  of  each  approach  must  be 
examined  for  each  data  element. 

3.  Identify  data  shortfalls:  When  all  the  required  data  are 
already  being  accurately  collected,  the  missing  data  (data 
shortfalls)  must  be  addressed.  If  the  data  shortfalls  can  not  be 
added  to  the  data  collection  effort,  other  techniques,  such  as 
comparability  analysis  or  computer  simulation,  should  be  used. 
Data  with  questionable  reliability  or  validity  puts  the  resuts  of  any 
analysis  in  question. 

4.  Structure  the  data  collection  forms:  Prior  to  the  data 
collection,  the  data  collection  forms  need  to  be  structured  to  allow 
for  the  easy  recording  of  the  data.  Easy  to  use  forms  allow  the  data 
collector  to  concentrate  on  the  accuracy  of  the  data  instead  of  the 
recording  of  it. 

5.  Define  data  collection  procedures:  The  procedures  that  the 
data  collectors  must  follow  should  be  defined  in  enough  detail  to 
cover  expected  situations.  These  procedures  should  then  be  tested  in 
a  simulated  environment  and  refined  if  necessary.  The  more 
situations  that  are  covered  the  more  reliable  the  data  collection 
process. 
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6.  Identify  the  data  collection  skills  required:  The  kind  of 
data  being  collected  should  help  determine  the  necessary  skills  of 
the  data  collectors.  For  example,  damage  repair  data  should  be 
collected  by  a  maintenance  technican  not  a  logistics  specialist  or 
computer  analyst.  Understanding  the  job  is  important  part  of 
accurate  data  collection. 

7.  Train  data  collectors:  The  data  collectors  should  be 
trained  on  how  to  use  the  data  collection  forms  or  data  entry 
devices.  The  ease  of  data  entry  increases  the  accuracy  of  data 
recording. 


Normalizing  the  Data 

Normalization  of  data  changes  the  data  from  one  condition  to 
another  desired  condition.  There  are  various  ways  to  normalize  data. 
Many  analysts  attempt  to  either  use  the  original  data  directly  or 
modify  the  original  data  to  the  desired  condition.  Either  works  well 
when  situations  are  similar.  However,  data  should  not  be  normalized 
simultaneously  over  several  conditions,  such  as  from  an  older 
aircraft  in  a  surge  war  to  a  current  aircraft  in  a  sustained  war. 
When  the  desired  condition  is  not  closely  related  to  the  original 
condition,  the  validity  of  the  data  normalized  becomes  questionable. 

To  normalize  multiple  conditions,  a  logical  stepwise  procedure 
must  be  used.  Using  the  same  conditions  mentioned  above,  the  first 
step  would  be  to  normalize  the  data  from  an  older  aircraft  in  a  surge 
war  scenario  to  the  same  older  aircraft  in  a  sustained  war  scenario. 
The  next  step  would  be  converting  the  normalized  parameters  from  the 
new  conditions  (older  aircraft  in  a  sustained  war)  to  the  current 
aircraft  in  the  sustained  war  (old  to  new  aircraft).  Step-by-step 
normalization  decreases  the  probability  of  errors  in  the  conversion. 
The  same  approach  can  be  used  to  normalize  data  from  other  scenarios 
such  as  from  the  Pacific  to  a  European  environment. 
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Data  Base  Considerations 


Care  must  be  exercised  when  using  data  from  a  foreign  source, 
because  it  will  have  different  characteristics  than  our  domestic 
data.  The  key  factors  to  consider  are:  (a)  the  types  and  lengths  of 
the  missions  flown;  (b)  the  physical  environment,  terrain,  and 
weather;  (c)  the  theater  of  operation;  (d)  the  use  of  combat  tactics 
such  as  strafing,  pop-up,  or  loft;  (e)  the  logistics  pipelines;  (f) 
the  maintenance  practices  and  tools;  and  (g)  the  manpower  strengths, 
including  the  reserves  and  industrial  base. 

A  simple  review  will  determine  if  differences  are  relevant. 
However,  if  a  significant  difference  does  exist,  common  comparability 
analysis  is  usually  the  simplest  and  most  accurate  method  of 
resolving  the  differences. 


III.  CONTRACT  MANAGEMENT  ISSUES 


Because  of  inconsistencies  with  the  contractor  progress  reports 
and  performance  and  cost  reports,  several  lessons  were  re-learned 
about  the  management  of  the  contract  that  may  apply  to  all  R&D 
contracts,  and  particularly  those  with  US  prime  and  foreign 
subcontractors.  The  contract  monitor  must  continually  monitor  all 
reports,  deliverables,  and  tests  for  compliance  with  the  Statement  of 
Work  (SOW). 

First,  the  contract  monitor  must  compare  the  progress  reports, 
cost  reports,  and  schedule  and  milestones  with  each  other.  Careful 
attention  should  be  given  to  relating  the  technical  progress  with  the 
schedule  to  ensure  that  the  contractor  is  on  schedule.  The  slightest 
delay  may  indicate  the  beginning  of  technical  problems  or  the  late 
delivery  of  the  Contract  Data  Requirements  List  (CDRL)  items.  Also, 
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the  technical  progress  should  be  compared  to  the  performance  and  cost 
report  to  determine  if  the  manpower  expended  is  appropriate.  If  more 
manpower  has  been  charged  to  the  effort  than  seems  appropriate,  the 
manpower  categories  should  be  thoroughly  examined.  The  contractor 
could  be  using  less  experienced  personnel  than  proposed.  The 
continued  use  of  less  experienced  personnel  may  extend  the  time 
necessary  to  complete  the  contract  or  may  result  in  an  inferior 
product.  Furthermore,  the  contractor  may  be  expending  more  manpower 
than  necessary,  because  the  technical  effort  is  more  difficult  than 
expected.  This  could  lead  to  an  unexpected  cost  overrun  if  not 
watched  closely.  Other  issues  also  need  to  be  reviewed.  For 
example,  travel  should  be  reviewed  to  determine  if  it  is  appropriate, 
required  by  the  contract,  and  scheduled  in  advance.  All  costs 
reported  should  be  continually  compared  to  the  technical  progress  and 
schedule.  Social  and  cultural  differences  with  foreign  companies 
makes  this  checking  process  very  difficult. 

The  contract  monitor  should  periodically  participate  in  an 
examination  of  all  charges  to  the  contract  at  the  contractor's 
facility.  The  contract  monitor  should  be  personally  involved  in  this 
financial  review,  because  there  are  many  charges  that  can  only  be 
identified  by  a  technical  person  with  an  understanding  of  the 
contract  requirements.  Actual  charges  may  include  specific  labor 
hours  for  each  individual  charging  to  the  contract,  equipment  charged 
to  the  contract,  computer  hours  used  by  type  of  computer,  and 
consultant  and  subcontractor  charges.  The  contractor  is  not  required 
to  show  the  actual  records  to  the  contract  monitor.  However,  the 
contract  monitor  can  request  through  the  Contracting  Officer  that  the 
Defense  Contract  Audit  Agency  (DCAA)  office  examine  the  contractor's 
actual  charges  on  the  contract  with  the  contract  monitor.  Periodic 
review  of  the  actual  charges  on  the  contract  for  their  relevancy 
reduces  the  possibility  of  mischarges  to  the  effort  and  a  cost 
overrun. 
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Finally,  the  contract  monitor  should  closely  follow  the  delivery 
schedule  to  ensure  that  all  CDRL  items  are  delivered  on  time.  This 
includes  periodic  reports  such  as  the  technical  progress  reports  and 
the  cost  and  performance  reports.  Continual  late  delivery  of 
periodic  reports  indicates  a  problem  and  should  be  reported  to  the 
Contracting  Officer.  Late  delivery  of  the  drafts  and  initial 
deliveries  of  major  CDRL  items  must  also  be  reported  to  the 
Contacting  Officer.  Ignoring  the  contracted  delivery  schedule  can 
result  in  the  loss  of  some  government  rights  to  the  deliverables  and 
authority  under  the  contract  to  take  action. 
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